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PREFACE 

'The  modern  control  and  programming  problems  studied  in 
this  memorandum  are  a  common  mathematical  formulation  of  situa¬ 
tions  that  arise  in  diverse  areas.  Common  examples  occur  in 
the  control  of  aircraft  and  missile  regimes,  control  of  reactors 
and  control  of  inventories.  These  problems  have  been  studied 
at  RAND  over  a  period  of  years  by  various  methods,  and  have 
been  the  subject  of  extensive  study  by  Soviet  mathematicians 
as  v/ell.  The  present  memorandum  studies  these  problems  from 
the  point  of  view  of  the  calculus  of  variations.  The  relation¬ 
ship  of  this  study  to  previous  RAND  work  and  some  of  the  Soviet 
work  is  discussed. 

This  paper  is  scheduled  to  be  published  in  The  Journal 
of  Mathematical  Analysis  and  Applications. 


SUMMARY 
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It  Is  shown  how  a  fairly  general  control  problem,  or  pro¬ 
gramming  problem,  with  constraints  can  be  reduced  to  a  special 
type  of  classical  Bolza  problem  In  the  calculus  of  variations. 
Necessary  conditions  from  the  Bolza  problem  are  translated  Into 
necessary  conditions  for  optimal  control.  It  la  seen  from 
these  conditions  that  Pontryagln's  maximum  principle  Is  a  trams-^ 
lation  of  the  usual  Weierstrass  condition,  and  Is  applicable 
to  a  wider  class  of  problems  than  that  considered  by  Pontryagln. 
The  differentiability  and  continuity  properties  of  the  value 
of  the  control  are  established  under  reasonable  hypotheses  on 
the  synthesis,  and  It  is  shown  that  the  value  satisfied  the 
Hamilton— Jacobi  equation.  As  a  consequence,  a  rigorous  proof 
of  a  functional  equation  of  Bellman  is  obtained  auid  Is  shown 
to  be  valid  for  a  much  wider  class  of  problems  than  heretofore 
considered.  A  sufficiency  theorem  for  the  synthesis  of  control 
Is  also  given. 
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VARIATIONAL  MSTHODS  IN  PROBLEMS  OP 
CONTROL  AND  PROORAMMINa 


1 .  INTRODUCTION 

A  controlled,  or  programmed,  system  Is  one  for  which  the 
state  at  time  t  Is  represented  by  a  real  n-dlmenslonal  vector 
x(t)  ■  (x^(t),  .  ..,  x‘’(t))  that  Is  determined  by  a  system  of 
differential  equations  and  Initial  conditions, 

(jt  "  ®  (^*^*'*)*  *  "  ^0*  1  "  1*  •••,  n, 

where  u  -  (u^(t),  u”'(t)).  The  m-dlmenslonal  vector  u(t) 

Is  called  the  control  function,  or  control,  or  the  program  for 
the  system;  It  Is  usually  required  to  satisfy  constraints 

(l«2)  R^(t,X,u)  0$  J  «  1,  ••*,  r« 


The  problem  of  optimal  control,  or  the  programming  problem.  Is 
to  choose  the  control  u(t)  so  as  to  bring  the  system  from  the 
given  initial  state  to  a  terminal  state  (t^,Xj^),  or  one  of  a 
collection  of  terminal  states  ((tj^,Xj^)),  In  such  a  way  as  to 
minimize  (or  maximize)  a  functional 


ti 

(1.5)  J(u)  »  g(tj,,Xj.)  +  f  f(t,x,u)dt. 


where  g  is  a  function  defined  on  the  set  of  terminal  states 
and  the  Integral  is  evaluated  along  the  solution  of  (1.1) 
corresponding  to  the  choice  of  u(t).  A  more  complete  and 
precise  statement  of  the  problem  will  be  given  In  Sec.  2. 
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It  is  generally  recegnized  that  In  the  absence  of  the 
constraints  (1.2),  control  problems,  as  usually  formulated, 
are  special  cases  of  the  problem  of  Bolza  in  the  calculus  of 
variations.  In  attacking  problems  in  which  constraints  of  the 
form  (1.2)  are  present,  as  well  as  constraints  of  the  form 

ti 

(1.4)  J  »''(t,x,u)dt  ^  C*',  k  -  1,  K, 

several  avenues  have  been  explored.  One  is  the  "maximum 
principle"  developed  by  Pontryagln  and  his  collaborators 
Boltyanskl  and  Oamkrelldze  (l>l  for  problems  of  the  following 
type.  The  constraints  are  Independent  of  x  and  require  u 
to  lie  in  a  closed  set,  the  function  g  is  absent,  the  terminal 
state  x^  is  a  prescribed  vector,  and  the  terminal  time  is 
arbitrary.  An  extension  of  the  maximum  principle  to  problems 
in  which  the  time  of  termination  is  fixed,  is  free, 

and  g  is  a  linear  function  of  the  coordinates  was  given  by 
Rozonoer  [14]. 

Another  approach,  which  is  formal  and  heuristic  in  charac¬ 
ter,  is  the  dynamic-programming  argument  of  Bellman  [l],  who 
presents  a  functional  equation  that  the  value  of  the  minimum 
as  a  function  of  initial  position  must  satisfy.  The  terminal 
condition  in  this  class  of  problems  has  t^  fixed  and  x^ 
free.  Rozonoer  [14]  has  rigorously  established  the  validity 
of  the  functional  equation  presented  by  Bellman  for  those 
problems  in  this  class  in  which 
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^  1  i 

g  -  z  CX(t.). 

A  different  set  of  techniques  has  been  used  In  dealing 
with  linear  systems  (1.1)  •  The  problem  of  determining  a  control 
u(t),  subject  to  the  constraints  |u^(t)|  ^  1»  1  -  1,  m, 

that  brings  x(t)  to  0  In  minimum  time  was  studied  for 
systems  with 


in  1  m  1 

0^-2  aiiX«^  +  2  b  .uJ 

J-1  J-1 

by  Bushaw  [5],  Bellman,  ailcksberg,  and  Gross  [2],  and  Gamkrelldze 
[6].  The  problem  of  determining  u  so  as  to  minimize  the  time 
required  for  x(t)  to  hit  a  moving  particle  z(t)  for  linear 
systems  In  which  Sij  and  bij  are  functions  of  time  was 
studied  by  Krasovskil  [8]  and  LaSalle  [10].  The  paper  by 
LaSalle  gives  a  brief  survey  of  the  other  papers  cited  In  this 
paragraph.  Krasovskil  [9]  has  considered  the  last  problem  for 
systems  (1.1)  of  the  form 


0^  .  f^(t,x)  +  b^(t)u. 


In  the  first  part  of  this  paper  we  shall  show  how  a 
fairly  general  control  problem  with  constraints  can  be  reduced 
to  a  special  type  of  classical  Bolza  problem.  Necessary 
conditions  from  the  Bolza  problem  will  be  translated  into 
necessary  conditions  for  optimal  control.  These  conditions 
give  more  Information  than  the  necessary  conditions  pfesented 


RM-2888 

4 


by  the  authors  cited,  and  are  applicable  to  wider  classes  of 
problems.  For  example,  it  will  be  seen  that  the  maximum 
principle  is  a  restatement  of  the  Welerstrass  condition  in  the 
calculus  of  variations  and  is  applicable  to  more  general  problems 
than  those  considered  in  [I3]  and  [l4].  Results  on  "bang— bang" 
control  can  be  derived  from  Corollaries  1  and  2  of  Theorem  2, 
but  we  shall  not  develop  this  topic  here. 

Theorem  2  of  the  present  paper,  which  is  the  main  theorem 
concerning  the  necessary  conditions,  was  stated  in  slightly 
different  form  by  Hestenes  [7]  in  connection  with  aircraft 
climb  problems,  but  was  never  published  by  him.  Because  of 
the  relative  unavailability  of  (7],  we  shall  present  the  proof 
of  Theorem  2.  The  constraint  conditions  of  the  present  paper 
are  slightly  different  from  those  of  Hestenes.  We  also  consider 
the  case  of  discontinuous  f,  0^,  and  R^,  and  give  simple 
criteria  for  normality  in  a  special  class  of  problems. 

In  the  second  part  of  the  paper  we  study  the  function 
W(t,x),  which  is  defined  as  the  value  of  the  minimum  (or 
maximum)  of  (I.3)  as  a  function  of  initial  position.  We  detez^ 
mine  the  differentiability  prapertles  of  w  under  reasonable 
assumptions  on  the  synthesis  of  control,  and  show  that  in  its 
regions  of  differentiability  the  function  w  satisfies  the 
Hamilton-Jacobi  equation.  By  combining  this  equation  with  the 
Welerstrass  condition  (or  maximum  principle),  we  can  rigorously 
establish  the  functional  equation  of  Bellman  [l]  and  obtain  a 
statement  about  its  regions  of  validity  for  a  vory  general 
class  of  Problems. 
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Our  last  theorem  la  a  sufficiency  theorem  that  la  useful 
In  synthesizing  the  control.  This  theorem  Is  a  varla.  t  of  the 
standard  sufficiency  theorem  In  the  calculus  of  variations.  A 
similar  theorem  was  stated  by  Brealcwell  [4];  his  statement* 
however,  needs  an  additional  hypothesis  to  be  valid,  and  his 
proof  la  formal. 

We  conclude  our  Introductory  remarlcs  with  the  observation 
that  problems  In  which  constraints  of  the  form  (1.4)  are  present 
can  be  reduced  to  problems  without  these  constraints  by  the 
Introduction  of  new  state  variables  and  aoBOclated  Initial  and 
terminal  conditions  as  follows: 


-  ♦“(t.x.u),  .  0,  x-'+Nti)  1  0*', 


1 ,  . . . ,  K . 


2.  NOTATION  AND  STATEMENT  OP  PROBLEM 

Vector  matrix  notation  will  generally  be  used.  Vectors 
and  matrices  will  be  denoted  by  single  letters.  Superscripts 
will  be  used  to  denote  the  components  of  a  vector;  subscripts 
will  be  used  to  distinguish  vectors.  Vectors  will  be  written 
as  matrices  consisting  of  either  one  row  or  one  column.  We 
shall  not  use  a  transpose  symbol  to  distinguish  between  the  two 
usages,  as  It  will  be  clear  from  the  context  how  the  vector  Is 
to  be  considered.  If  A  Is  a  matrix  of  m  rows  and  n 
columns,  x  Is  an  m-dlmenslpnal  vector,  and  y  Is  an 
n-dlmenslonal  vector,  then  In  the  product  xA,  x  must  be  a 
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row  matrix,  and  In  the  product  Ay,  Y  must  be  a  column  matrix. 
Thus  we  write  the  Inner  product  of  two  vectors  X  and  y 
simply  as  xy;  a  quadratic  form  with  matrix  A  we  write  as 
xAx. 

The  operator  (d/dt)  will  generally  be  denoted  by  a  prime. 
Thus,  the  system  (1.1)  will  be  written  as 

(1.1)  x'  -  a(t,x,u),  x(tQ)  -  x^, 

and  the  constraints  (1.2)  as  R(t,x,u)  >  0.  (A  vector  Is  non- 
negative  If  and  only  If  every  component  Is  nonnegative.)  If 
Z(t,x,u)  Is  a  vector— valued  function  that  Is  differentiable 
on  a  region  of  (t,x,u)  space,  we  denote  the  matrix  of 
partial  derivatives  (^zV^^c*^)  by  Z^;  the  symbol  Z^  has 
similar  meaning.  For  real— valued  functions  z(t,x,u),  the 
symbols  and  Z^^  represent  vectors  of  partial  derivatives. 

We  denote  the  determinant  of  a  square  matrix  A  by  1|a11. 

Let  8^  bounded  region  of  (n  +  1)— dimensional  (t,x) 

space  and  let  be  a  region  of  m-dlmenslonal  u  space .  Let 

Let  V  be  a  manifold  of  class  C",  of  dimension 
P  1  lying  In  <0.  and  given  parametrically  by  equations 

(2.1)  t  -  tj(a),  X  -  Xj(0), 

where  a  -  (a^,  ••.,  a^)  ranges  over  an  open  cube  ^In 
p-dlmenslonal  space.  Points  of  will  henceforth  be  denoted 
as  (tj_,Xj^);  we  shall  call  ^  the  terminal  manifold.  Let 
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f(t,x,u)  be  a  real— valued  function  of  class  C”  on  »  let 
g(a)  be  a  real— valued  function  of  class  C'^  on  and  let 

the  vector-valued  functions  a{t,x,u)  ■  (O^,  o”)  of  (1.1) 

and  R(t,x,u)  «  (r^,  r^)  of  (1.2)  be  of  class  C"  on  , 

Furthermore,  let  the  constraint  vector  R  satisfy  the  follow¬ 
ing  constraint  conditions; 


(2.2) 


(1)  If  r  >  m,  then  at  each  point  of 
components  of  R  can  vanish. 


J 


at  most  m 


(11 )  At  each  point  of  the  matrix  (  Sr^/^u*^  ) ,  where 

1  ranges  over  those  Indices  such  that  R^(t,x,u)  ■  0 


and  J  ■  1, 


m, 


has  maximum  rank. 


Consider  the  class  of  all  functions  u  »  u(t)  that  are 
piecewise  C'  (l.e.,  each  component  u^  of  u  Is  piecewise 
continuous  and  has  piecewise  continuous  first  and  second 
derivatives)  on  the  closure  of  the  projection  of  on  the 
t  axis,  and  have  range  contained  In  -a.  For  each  such  u  we 
can  obtain  a  continuous  solution  of  (1.1)  that  defines  a  curve 
K,  with  possible  comers.  In  jy.  Let  be  the  subclass  of 
this  class  of  functions  u  with  the  following  properties: 

(i)  The  curve  K  Is  defined  and  Is  Interior  to  d& for 

^0  ^  ^  ^  ^1"  where  (t3^,Xj^)  =  Is  a  point 

of  and  K  does  not  Intersect  c/  for  any 

(11)  Along  K,  the  constraints  (1.2)  are  satisfied;  l.e., 
R(t,x(t),u(t))  ^  0* 
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The  class  which  depends  on  is  called  the  class 

of  admissible  controls.  For  a  given  (t^^XQ)  It  may  be  void. 

The  problem  of  optimal  control  Is  to  find  an  element 
u  €  ^  that  minimizes  (or  maximizes)  the  functional 


(2.5) 


J(u) 


f (t,x,u)dt 


over  all  u  c  66,  where  the  Integral  Is  taken  along  the  curve 
K  corresponding  to  u,  and  a  Is  the  parameter  value  associ¬ 
ated  with  (tj^jX^)  ■  (t^,x(t^)).  For  definiteness  we  shall 
henceforth  assume  that  (2.5)  Is  to  be  minimized. 

We  note  that  the  problem  of  optimal  control  as  presented 
here  Is  equivalent  to  the  problem  In  which  g  »  0  or  the 
problem  In  which  f  ■  0.  The  equivalence  of  these  problems 
can  be  shown  by  making  transformations  similar  to  those  used 
to  show  the  equivalence  of  the  problems  of  Bolza,  Lagrange,  and 
Mayer  In  the  calculus  of  variations  ([5)»  PP»  189—190). 


5»  THE  EQUIVALENT  BOLZA  PROBLEMS 

Let  y  «  (y^,  ...,  y"*)  be  an  m-dlmenslonal  vector.  To 
the  system  (1.1)  adjoin  the  system  of  differential  equations 


(5.1)  y'  -  u,  y(tQ)  -  0. 

The  following  problem  of  Bolza  in  (n  +  m  +  1) -dimensional 
(t,x,y)  space  with  differential  inequalities  as  added  side 
conditions  Is  clearly  equivalent  to  the  problem  of  optimal 
control  posed  in  See.  2. 
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Problem  I.  Find  an  arc  (x(t)»y(t))  that  minimizes 

(5-2)  g(a)  f(t,x,y')dt 

^0 

In  the  class  of  arcs  that  are  piecewise  C",  that  satisfy  the 
differential  equations 

(3-3)  0(t,x,y')  -  x'  -  0, 

the  differential  Inequalities 

(3.4)  R(t,x,y')20* 

and  the  end  conditions 

xCt^)  -  Xq,  y(tQ)  *  yQ  -  0, 

(3.3) 

»  y(tj^)  »  n. 

where  r\  ■  (il^,  ,  t}*"). 

By  means  of  a  device  used  by  Valentine  in  [l5)j»  we  obtain 
the  following  problem  of  Bolza,  which  has  no  Inequality  side 
conditions  eUid  Is  equivalent  to  Problem  I. 

Problem  II.  Find  an  arc  (x(t),y(t),^(t ) ),  where 
I  .  ...,  that  minimizes  (3-2)  in  the  class  of  arcs 

that  are  piecewise  C",  that  satisfy  the  differential  equations 
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G(t,x,y ’ )  -  x'  »  0, 

(5.6) 

R(t,x,y' )  -  iVr  -  0> 
and  the  end  conditions  (5.5)  and 


(5.7)  ?(tQ)  »  ^0  -  0>  ii^i)  »  Cl  »  T, 

where  t  »  (t^,  t^)  and  )^  =  f ,  ...,  (4^  )^). 

Let  u*e^^be  an  optimal  control,  let  K*  be  the  corre— 

*  * 

spending  curve,  and  let  x  (t)  be  the  function  defining  K  , 

for  tQ  <  t  <  tj^.  Let  y  (t)  denote  the  solution  of  (5*1) 

* 

for  u  ■  u  .  It  follows  from  the  preceding  discussion  that 
(x*(t ),y*(t ) )  satisfies  Eqs.  (5.5)  -  (5.5)  and  minimizes  (5.2)* 
Hence  the  arc  defined  by  (x  (t),y  (t),^  (t)),  where 

(1  (t)')^  -  H(t,x  ,y  ’)#  C  (tQ)  -  0, 


fumlshes  a  minimum  for  Problem  II.  We  denote  this  arc  by 
«  #***'*i*i 

We  assert  that  at  every  element  (x  ,y  ,x  ,y 

of  Kg,  the  equations  (>.6)  aj?e  Independent;  that  is,  the 

matrix 


) 


/  \  * 
has  rank  (n  +  r)  along 


where  I  is  the  n-^dinensional 
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identity  matrix  and  2  H'  Is  an  r  x  r  diagonal  matrix  with 
entries  2(^^)'  on  the  diagonal,  1  ■  1,  r.  In  order  to 

prove  the  assertion  we  first  suppose  that  the  first  r^  rows, 

0  <  r,  <  r,  of  the  submatrix  (R  ,  0  —  2  2')  have  elements 

2  /  0,  and  the  remaining  rows  have  elements  2  -0. 

This  can  always  be  achieved  by  permuting  rows  and  relabeling. 
The  matrix  (3.8)  now  has  the  form 


where  D  Is  an  (n  +  r^.)  by  (n  +  r^)  diagonal  matrix  with 
nonzero  entries  on  the  diagonal  and  0  Is  a  zero  matrix.  The 
matrix  consists  of  the  last  r  -  r^  rows  of  the  matrix 

Ry,.  For  each  of  these  rows,  we  have  (?/)'  »  0.  Consequently, 

we  have  R  (t,x  ,y  ')  »  0,  1  »  +  1,  r.  Prom  (5.1)  we 

see  that  this  Is  equivalent  to  R  (t,x  ,u  )  =  0,  for 
1  ■  r^^  +  1,  ...,  r.  Prom  the  constraint  conditions  (2.2)  It 
follows  that  r  —  r^  <  m  and  that  the  matrix  with  elements 
dR^/du'^,  1  »  r^  +  1,  ... ,  r,  J  »  1,  . . . ,  m,  has  rank  r  —  r^ 

for  (t,x  ,u  )  along  .  Hence,  It  follows  that  A2  has 

rank  r  -  r^  and  (3.8)  has  rank  (n  +  r^)  +  (r  -  r^)  ■  n  +  r, 
as  required. 

The  above  argument  actually  is  not  restricted  to  K^;  It 
shows  that  (3*8)  has  rank  (n  +  r)  at  all  elements 
(t,x,y,  e^,x' ,y ' ,  e, '  )  for  which  (3.6)  holds. 
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4.  NECESSARY  CONDITIONS  FOR  PROBLEM  II 

Since  K2  furnishes  a  minimum  for  Problem  II  and  the 
matrix  (3*8)  has  rank  (n  r)  wherever  Eqs .  (5*6)  hold.  It 
follows  (Bliss  [3]*  McShane  [ll])  that  the  following  necessary 
conditions  hold  along  : 

Theorem  1 .  There  exist  a  constant  \q  >  0,  n— dlmen-r 

slonal  vector  X-(t),  and  an  r— dimensional  vector  M.(t), 
defined  on  the  Interval  <  t  <  t^,  such  that  (kQ,X(t  ),M.(t) ) 
Is  never  zero  and  such  that  X(t)  and  M-(t)  are  continuous, 
except  perhaps  at  values  of  t  corresponding  to  corners  of 
K2  >  where  they  possess  unique  right-hand  and  left-hand  limits  * 
Moreover,  the  function 


(4.1)  P(t,x,y,^,x',y',e',XQ,X,M.) 


X^f  +  X(G  -  X' )  +  4(R  - 


* 

satisfies  the  following  conditions  along  : 

(1)  (Euler-Lag range  equations)  Between  corners  of  Kg, 
we  have 


(4.2) 


^  dPy,  ^  dPg, 

dt  “  ^x*  dt  *  ^y*  dt 


At  a  comer,  these  equations  hold  for  the  unique  one— sided  limits. 

(la)  (welerstrass-Erdmann)  At  a  corner  of  Kg,  P^,,  Py.,, 

p^,,  and  (P  -  x'  P^,  -  y'  Py,  -  f  P^i)  have  well-defined, 
one-sided  limits  that  are  equal. 
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(11)  (Transversallty ) 

A1 

the  end  point 

( , X* ( t ) , y j ( t ) , C 1 ( t ) )  of 

« 

Kg, 

we  have 

(4.3)  (P  -  X'  P^,  -  y'  Fy, 

-  ? 

'‘lo  ^x'*ao  ^0*9  ■ 

^y'  ^iTi  "  0' 

,  -  0. 

(111)  (Welerstrass )  For  all  (t,x, , Y' , C' )  / 

(t,x, ^,y,x' ,y ' )  and  satisfying  (5*6),  the  Inequality 

(4.4)  ^(t,x,y,e,x',yV,C',X',y',C',lo>>-,ti)  >  0 

holds j  where 

5-  P(t,x,y,e,X*,Y',C' )  -  P(t,x,y,^,x'>y',e' ) 

-  (X'  -  x')p^,  -  (Y*  -  y')Py,  -  (C  -  e')P^i. 

I 

the  functions  ,  and  P^ ,  being  evaluated  at 
(t,x,y,^,x',ySt',XQ,X,^i),  and  the  arguments  (Xq,X,m.)  being 
omitted  throughout . 

(iv)  (Clebsch)  For  every  vector  (tt, p,k)  /  0>  where 

TT  —  {v  I  •  •  •  t  TT  )  )  P*  (p  t  ‘  >  P  )»  K  «  ( 1C  ^  j  K  )  ^ 

that  Is  a  solution  of  the  linear  system 

Oy,P  -  ITT  -  0, 

R  (  P  2£1*K  »  0  > 

iX 


(^.5) 
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the  following  Inequality  holds: 

(4.6)  T  »■  +  P  ^y'y'P  ”  ^  ^  0. 

5.  NECB33ARY  CONDITIONS  FOR  PROBLEM  I 

We  now  follow  Valentine  [15]  and  translate  the  necessary 
conditions  for  Problem  II  Into  necessary  conditions  for 
Problem  I.  We  first  consider  the  Euler  equations.  Prom  (4.1) 
we  get 

(5.1)  P*  -  0,  P  . ,  -  -  1-1*  ...,  r. 

^  r 

Hence  It  follows  from  the  third  equation  In  (4 .2 )  that 

4  4  ^ 

d(M.  ^  )/dt  -  0  along  Kg.  This  and  the  continuity  of  P^, 

*  11'  * 
at  comers  of  Kg  Imply  that  u  ^  Is  constant  along 

Prom  the  transversallty  condition  (4.3)  we  get 

from  (3.7)  we  obtain  -  I,  where  I  Is  the  r  x  r 

Identity  matrix.  Therefore,  we  have  P^,  -  0  at  the  right— 

*  11'  * 
hand  end  point  of  Kg,  and  consequently  u  i  -  0  along  Kg. 

It  now  follows  from  the  second  equation  In  (3*6)  that  along 

* 

Kg, 

(5.2)  -  0  1  -  1,  ...,  r. 

A  similar  argument  shows  that  along  YL,  we  have  p  ,  -  0. 

^  y 

We  now  Introduce  the  function 


(5.3)  H(t,x,y ' , Xq, X)  —  Xq  f(t,x,y')  +  XQ(t,x,y' ) 
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Clearly, 


(5.4)  P  -  H  -  Xx'  +  tt(R  -  . 


The  following  are  Immediate  consequences  of  (5.4); 


(5.5) 

p  ,  »  -  X. 

x‘ 

« 

Since  Py,  -  0  along  K^,  we  see  that  along 

(5.6)  Hy,  +  HRy,  -  0. 


«2 


we  have 


Prom  (4.2)  and  (5.5)#  along  we  also  have 


(5.7)  X'  .  -  +  ixR^). 


It  follows  from  the  vanishing  of  Py,  and  P^,  along 
Kgj  and  from  (5.4),  (5.5),  and  the  second  equation  of  (5.6), 
that 

(5.8)  P  -  X'  -  y*  Py,  -  V  P^.  -  H 

along  K^.  Hence  It  follows  that  the  transversallty  condition 
becomes 

(5.9)  +  Htj^  -  Xxj^^  -  0. 

The  relationships  used  to  establish  (5.8)  and  the  fact 
that  (t,x,Y'>C')  aatlaflea  the  second  equation  of  (5.6) 
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enable  us  to  translate  the  Welerstrass  condition  (4.4)  Into 
the  condition 

(5.10)  H(t,x,Y',XQ,X)  ^  H(t,x,y',XQ,X). 

It  la  an  Immediate  consequence  of  (5.4)  that  (4.6)  becmnes 

V 

(5.11)  p((H  +  HR)„,„.)p  -  2  2  H^(x^)^  ^  0. 

^  ^  1-1 

If  >0  at  a  point  of  then  by  (5.2),  -  0.  If 

R^  -  0  at  this  point,  let  ir  -  0,  let  p  -  0,  and  let  k  be 
a  vector  with  1— th  component  1  and  other  components  0.  Then 
(tt, p,k)  /  0;  and  since  »  0,  (Tr,p,K)  Is  a  solution  of 
(4.5).  Hence  from  (5.11)  we  get  m-^  <  0  at  this  point. 
Consequently,  we  always  have 

1  * 

ix  <  0  along  Kg,  1ml,  r. 

Let  (t,x,y)  be  a  point  of  Kg  such  that  at  most  r^^. 
Where  r^^  <  m,  components  of  R(t,x,y')  vanish;  we  suppose 
for  definiteness  that  these  are  the  first  r^  components.  It 
follows  from  (2.2)-(il)  that  the  system  of  linear  equations 

y  oJ  _  ft  41 

Z  1 1  P  2  ?  K  »  0,  1  *  1,  ••  •  ,  r-i  # 

>1  ay*’  ^ 

1 

has  a  solution  In  p  and  k  -  (k  ,  ...,  k-  )  such  that  p  /  0 
and  K  -  0.  It  now  follows  from  the  second  system  of  equations 
In  (3.6)  and  the  assumption  that  R*’  >  0  for  J  >  r^,  that 
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the  system  (4.5)  has  a  solution  {p,Tr,K)  such  that  p  /  0 
and  K  »  0.  Let  J  >  Since  indices  J  >  correspond 

to  components  >  0,  it  follows  from  (5*2)  that  -  0  for 
J  >  r^.  Hence  each  term  in  the  second  sturanation  in  (5.11) 
vanishes,  and  from  (4.5)  we  have 

(5.12)  p((H  +  HR)y,y,)p  >  0 

for  any  solution  vector  p  of  the  system 
m  4 

(5.13)  2  — rr  -  0,  1  ■  1,  r^. 

j-1  ay’  ^ 

The  conclusion  Just  stated  holds,  of  course,  even  if  m 
components  of  R  vanish.  In  that  case,  however,  the  system 

(5.13)  lias  only  the  trivial  solution. 

6.  NECESSARY  CONDITIONS  POR  THE  CONTROL  PROBLEM 

The  following  theorem,  in  which  necessary  conditions  for 
optimal  control  are  given,  is  an  immediate  consequence  of  the 
conclusion  obtained  in  Sec.  5,  and  of  the  use  of  (3.I)  to 
Jxistlfy  the  replacing  of  the  argument  y'  by  u,  whex^ver  y' 
occurs*  The  function  H  Is  now 

H(t,x,u,XQ,X)  -  XQf(t,x,u)  +  X0(t,x,u). 

Theorem  2 .  Let  u  e  be  an  optimal  control,  let  K 
be  the  corresponding  curve,  and  let  x*(t)  be  the  function 
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defining  K*  on  Then  there  exist  a  constant 

Xq  >  0,  an  n-dlmenalonal  vector  X(t)  defined  and  contlnuoua 

on  [Iq, and  an  r-dlmenslonal  vector  |i.(t)  <  0  defined 

and  continuous  on  the  Interval  except  perhaps  at 

* 

values  of  t  corresponding  to  corners  of  K  ,  where  It 
possesses  unique  right-hand  and  left-hand  limits >  such  that 
the  vector  (XQ^X(t))  never  vanishes^  and  such  that  the  following 
conditions  are  fulfilled; 

Condition  I.  Along  K*  the  following  equations  hold; 

(6.1)  x'(t)  - 

(6.2)  X'(t)  =•  -  +  MtRjj), 

(6.3)  ”u 

(6.4)  R^  =  0,  1  =  1,  . .  .>  r. 

At  the  end  point  (tj^,Xj)  of  K*  the  transversallty  condition 
holds ; 

(6.5)  Vo  +  «‘lo  -  ^10  - 

Along  K*,  the  function  H  Is  continuous. 

#  #  # 

Condition  II.  For  every  element  (t,x  ,u  )  of  K  and 
every  u  such  that  u  »  u(t)  for  some  u  in  CZ,  we  have 

(6.6)  H(t,x*,u,XQ,>-)  ^  H(t,x*,u*,XQA)- 
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Condition  III.  At  each  point  of  K*  let  R  denote  the 
vector  formed  from  R  by  taking  those  components  of  R  that 
vanish  at  that  point.  Let  e  -  (e^,  e”*)  be  a  nonzero 

solution  vector  of  the  linear  system  R^e  ■  0  at  a  point  of 
K*.  Then  e((H  +  M.R)uu)«  1  0  at  this  point. 

Equations  (6.1)  -  (6.4)  are  the  Euler  equations.  Condition 

II  follows  from  the  welerstrass  condition  (5*10),  and  Condition 

III  follows  from  the  Clebsch  condition  (5.12).  The  continuity 
of  H  along  K*  follows  from  the  continuity  of  the  left-hand 
member  of  (5.8)  along  K^,  and  the  continuity  of  X  follows 
from  (5.5)  and  the  continuity  of  P^,  (Welerstrass-Erdmann 
corner  conditions).  The  nonvanishing  of  (Xq,X)  along  K 

Is  established  as  follows.  If  (Xq,X)  were  zero  at  a  point  of 
K*,  then  from  (6.5)  we  would  have  uR^  ■  0  at  this  point. 

For  the  sake  of  definiteness,  suppose  that  the  Indexing  Is  such 
that  R^  »  0  for  1-1,  ...,  r^^,  where  by  (2.2),  r^  <  m. 

Hence,  by  (6.4),  we  have  »  0  for  1  >  r^^.  Thus  the 
condition  |xR^  =  0  reduces  to  a  system  of  linear  equations  In 
p.^,  ...,  P  ^  with  coefficient  matrix  (^rV^u^)*  i  ■  •••> 

k  =  1,  .. .,  m.  Prom  (2.2)  -  (11 ),  this  matrix  has  rank  r^. 

Hence  p  =  . . .  =  p  ■  0  Is  the  only  solution  of  the  linear 
system.  Thus,  we  have  shown  that  if  (X*^,X)  la  zero  at  a 
point,  then  the  vector  (x'^,X,p)  must  also  be  zero,  contradicting 
the  assertion  of  Theorem  1, 
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If  the  constraints  are  specialized,  then  Important 
simplifications  can  be  effected  in  the  Euler  equations. 

Corollary  1.  Let  the  constraints  be  of  the  form 


B^(t,x)  <  u^  <  A^(t,x), 


where  and  each  A^  and  B^  is  Of  class  C"  on  > 


Then  at  each  point  of  K  we  have 


H 


i 


>  0  if  u*^  -  B^, 

1  *1  1 
-  0  i  f  B^  <  u  <  A  , 

<  0  if  u*^  »  A^, 


1  ^  ^  m  • 


If  we  write  the  constraints  as  A^  -  u^  2  0  and 
u^  —  B^  ^  0,  i  a  1,  m,  we  obtain  a  2m-<31menslonal  con¬ 

straint  vector  with  components  A^  —  u^  and  u^  -  B^.  It 
follows  from  the  condition  A^  >  B^  and  the  form  of  the  con¬ 
straints  that  (2.2)  is  satisfied.  The  conclusion  of  the 
corollary  follows  from  (6.3)  and  (6.4)  by  straightforward 
calculation  and  use  of  the  condition  ix  <  0. 

Remark .  If  the  1— th  component  of  u  is  constrained  only 

from  one  side,  say  u^  <  A^(t,x),  then  H  ^  ■  0  if  u^  <  A^ 

1  i 

and  H  <  0  if  u  ■  A  .  Similar  statements  hold  fpr 

^  B^. 

Another  Important  special  case  is  one  in  which  the  con¬ 
straints  are  independent  of  the  state,  that  is,  R(t,x,u)  s 
R(t,u).  since  Rjj  ■  0  in  this  case,  we  have  the  following 
corollary; 
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Corollary  2.  If  R  Is  Independent  of  x,  then  equation 

(6.2)  becomes 

(6.2) '  x«  -  ^ 

In  the  problem  considered  by  Pontryagln  [I3],  the  con¬ 
straints  required  u  to  lie  In  a  fixed  closed  set.  Independent 
of  time  t  and  position  x.  Equations  (6.1),  (6.2)',  and 
(6.6)  constitute  the  maximum  principle  as  stated  by  Pontryagln. 
Our  function  H  Is  the  negative  of  Pontryagln ' s ,  so  that  his 
maximum  appears  as  a  minimum  In  our  paper.  Note,  however,  that 
the  Euler  equations  and  Condition  II  of  Theorem  2,  which  Is  the 
Welerstrass  condition,  give  a  minimum  principle  for  a  wider 
class  of  problems. 

Remark.  Note  that  If  the  and  of  Corollary  1  are 

constants,  then  the  results  of  both  corollaries  are  valid. 

7.  INTEGRABLE  CONTROLS 

Instead  of  considering  functions  u  ■  u(t)  that  are 
Piecewise  C",  we  can  consider  functions  that  are  merely 
assumed  to  be  Lebesgue  Integrable.  In  this  way  we  can  define 
a  class  of  admissible  controls  (X  ^ >  and  we  can  look  for  an 
optimal  control  u*  In  a^.  The  curves  K  corresponding  to 
functions  u  In  will  be  defined  by  absolutely  continuous 

functions  x(t),  and  so  will  be  rectifiable.  We  can  reduce 
the  control  problem  with  constraints  to  a  Bolza  problem  Wltl>- 
out  constraints  as  we  did  before,  except  that  the  functions 
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(x(t),y(t), ^(t) )  are  now  absolutely  continuous.  To  this 

problem  we  can  apply  a  theorem  of  McShane  (Theorem  l6.1,  [12]). 

We  can  then  translate  back  to  the  original  control  problem  and 

obtain  the  result  that  the  conclusions  of  Theorem  2,  appropriately 

* 

modified,  hold  almost  everywhere  along  a  curve  K  corresponding 
to  a  control  u*  that  minimizes  (2*3)  over  all  u  In  , 

8 ..  NORMALITY 

A  piecewise  C"  minimizing  curve  K  ,  or  equivalently 

the  corresponding  curve  of  Problem  II,  Is  said  to  be 

normal  If  there  are  no  sets  of  multipliers  with  »  0.  (See 

[3]*  PP«  213-219.)  If  the  minimizing  curve  Is  normal,  then  the 

multipliers  can  be  chosen  so  that  *  1,  and  with  this  choice 

of  Xq  they  are  unique.  If  the  curve  is  not  normal,  there 

may  be  no  neighboring  curves  that  satisfy  the  differential 

equations,  constraints,  and  end  conditions.  Necessary  and 

sufficient  conditions  for  normality  are  given  In  [3].  These 

criteria  applied  to  the  present  problem  would  Involve  variations 
* 

along  K2  and  would  generally  be  difficult  to  apply  in  practice. 

We  shall  give  a  condition  for  normality  In  the  control  problem 

that  Is  sufficient,  but  not  necessary.  It  Is,  however,  easier 

to  apply  In  practice,  and  reduces  to  a  very  simple  condition 

in  the  special  case  that  the  terminal  manifold  Is  n-dlmenslonal 
#  #. 

At  (t2^,Xj^),  the  end  point  of  K  ,  let  r^^  components 

#  # 

of  R(tj,x^,u  (tj_))  vanish.  Prom  (2.2)  —  (i)  we  get  <  m. 

A 

Let  R  denote  the  r^^-dlmenslonal  vector  formed  from  R  by 
taking  those  cdmponenta  of  R  that  Vanish  at  (t^^x^),  and 
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let  M-  be  the  vector  formed  from  p,  by  taking  the  corresponding 
components.  Then  from  (6.4)  we  have  M.'^(t2)  »  0  for  those 

A 

components  of  p  that  are  not  In  p.  Let  M  denote  the  n 
by  p  matrix  with  typical  element 


(8.1) 


dt, 


i*  • • • f  J  ■  1 >  • • • $  p 


* 

where  the  elements  are  evaluated  at  the  end  point  of  K  .  Let 
C  denote  the  (n  +  r^^)  by  (m  +  p)  matrix 


where  and  ^  are  evaluated  at  (t^,x^). 

* 

If  K  is  not  normal,  then  there  exists  a  set  of 

multipliers  (X^, X,p)  with  X^  ■  0.  Prom  (6.3)  smd  (6.5)  we 

#  # 

see  that,  at  the  end  point  (tj^,x^)  of  K  ,  the  vector 
(X,p)  Is  a  Solution  of  the  linear  system  (X,p)C  -  0.  The 
following  theorem  Is  now  a  consequence  of  a  standard  theorem 
concerning  the  solutions  of  homogeneous  linear  systems  and  the 
fact  that  (X,p)  cannot  be  zero  If  X^  »  0: 

Theorem  3.  If  the  rank  of  C  Is  (n  +  r^),  then  K 
Is  normal . 

Note  that  C  can  have  rank  (n  +  r^)  only  when 
(n  +  r^)  <  (m  +  p),  and  that  Theorem  3  Is  not  a  necessary 
condition* 
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Corollary.  If  </  is^  n-dlmenslonal  and  K  la  not  tangent 
^  * 

to  c/  t  then  K  la  normal. 

If  x/  is  n-dlmenalonal,  the  matrix  M  Is  an  n  by  n 
matrix.  By  (2.2)  -  (11),  the  r^  by  m  matrix  ^  haa  rank 

r^.  Hence,  c  has  rank  (n  +  r^^)  whenever  M  has  rank  n* 

*  ^ 

Since  K  Is  not  tangent  to  c/  ,  the  matrix 


has  rank  n  +  1.  If  for  each  j  -  1,  ...,  n  we  multiply  the 
first  column  of  this  matrix  by  -  and  add  the  result 
to  the  J-th  column,  we  get  the  matrix 


G  -  M 


Hence  M  has  rank  n  and  the  corollary  follows. 


9.  DISCONTINUOUS  f,  0,  R 

Let^?l.be  a  manifold  of  dimension  n,  lying  In  and 

dividing  o^^into  two  regions,  such  that  some  or  all  of  the 
functions  f,  0,  and  R  are  discontinuous  across  Let 

the  discontinuity  of  a  function  be  such  that  the  function  and 
Its  derivatives  have  unique  one-sided  limits.  Further,  let 
us  aBsums  that  K*  intersects  at 
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and  Is  not  tangent  to  y/L  at  this  point .  It  can  be  shown  by 
appropriate  modifications  of  the  arguments  In  [3]  (pp»  196-202) 
that  the  multipliers  X  and  u  of  Problem  II  need  not  be 
continuous  at  t  ■  t^^,  but  will  have  unique  right-hand  and 

left-hand  limits  at  will  P  and  Its  various 

* 

partial  derivatives  when  evaluated  along  K^.  Although  P^,, 
Py,,  and  P^,  need  not  be  continuous  across  the 

expression 

(P  -  x'  P^,  -  y'  Py,  -  P^,)dt  +  +  Pyi<iy2 

has  equal  right-hand  and  left-hand  limits  along  at  (tg^Xg) 

for  all  differentials  dtg,  dXg  on  7^  and  all  dyg  and  d^g. 
For  the  original  control  problem  this  translates  Into  the 
condition  that 

(9.1)  (H'’’  -  H“)dtg  -  (X'*’  -  X”)dXg  -  0 

at  (tgfXg),  where  the  one-sided  limits  are  evaluated  along 
♦ 

K  . 

10.  DEPINITION  OP  SYNTHESIS 

Consider  a  point  (tj,Xj^)  of  the  p-^lmenslonal  terminal 
manifold  ,  where  0  <  p  <  n.  Let  7!^'  denote  a  region  in 
(n  —  p)— dimensional  space  over  which  a  vector  ranges.  If 
p  ■  n,  then  <p  is  the  zero  vector  .  Let  u*  be 

a  function  defined  In  some  Interval  [tQ,  t^^l,  where 
to  »  tQ(tj,  ,X3_><p)»  Such  that  the  following  condition  holds; 
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Assumption  1.  (i)  The  function  u  Is  piecewise  C" 

on  [tQ,tj^],  and  Its  range  lies  In  (11)  If  u*  Is 

substituted  into  (6.1)  (or  equivalently  Into  (1.1)),  the 
resulting  differential  equation 

(10. 1)  x'  »  G(t,x,u  ( t i t^, cp) ) ,  x(t-j^)  ■ 

has  a  continuous  solution  x  (t;t2»x^,(p)  on  (tQ,tj]  such 
that  (t,x  )  lies  In  and  R(t,x  ,u  )  ^  0. 

We  denote  the  curve  corresponding  to  x  (t;t j,Xj^,q>)  by 
K(tj^,X2^,9)  • 

We  now  suppose  that  the  assumptions  Just  made  for  a 
particular  point  (tj^,Xj^)  hold  for  all  points  (tj^,X2)  of 
.  Prom  (2.1),  we  have  (t2^,Xj^)  ■  (tj^(a),Xj^(d) ),  where 
ranges  over  an  open  cube  ^  in  a  p-dlmenslonal  space.  Let  9 
be  an  n-dlmenslonal  vector  defined  as  follows; 

(10.2)  0  «  (a,<p),  ae  q>  c 

We  define  functions 

*  to(tj^(a),  X3_(g),(p), 

(10.5) 

t^(fl)  »  tj(a). 


and  functions 


rm-2888 

27 


(10.4) 

x*(t,8)  =  x*(t;t3^(o),x^(0),(p), 

for  a  In  ^  in  and  tQ(S)  <  t  <  t^(9).  The 

differential  equation  (10.1)  can  now  be  written  as 

(10.5)  x'  =  O(t,x,u*(t,0)),  x(tj^(e))  -  Xj(a). 

Clearly,  x*{t,&)  is  a  solution  of  (10.5).  We  shall  denote 
the  curve  K(tj^,x^,(F)  by  K(9). 

Let  0.  denote  the  domain  of  definition  of  u  (t,0)  and 
X  (t,0);  that  is,  the  set  of  points  (t,S)  in  (n  +  1)— 
dimensional  space  with  6  as  in  (10.2)  and  tQ(0)  <  t  <  tj^(0). 
Clearly,  Q  has  nonvoid  Interior,  which  we  denote  by  It 

follows  from  (2.1)  and  (10.3)  that  defines  a  C” 

manifold  of  dimension  n  in  (t,©)  space  and  that 

is  part  of  the  boundary  of  fi.  We  also  suppose  that 
defines  a  C”  manifold  of  dimension  n. 

A  set  of  functions 

t  =  tj^(©),  1  *  1,  2,  ...,  a» 

defined  and  C'  on  the  region  defined  in  (10.2),  with  tj(©) 
as  in  (10.3)  and  such  that 


to(®)  <  ^1+1^^^  ^  ^  *  1*  .  .  .> 


a  1, 
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will  be  said  to  Induce  a  regular  decomposition  of  Clearly, 

each  t^(6),  i  >  2,  defines  a  C"  manifold  of  dimension 

n  lying  in  oP .  We  let  denote  the  manifold  defined 

by  tQ(6).  We  define  subregions  of  n®  as  follows: 

-  E{(t,e)  €  n°|t^^3_(e)  <  t  <  t^(e)),  i  -  i,  a  -  i. 

We  shall  say  that  a  function  h(t,6)  Is  piecewise  on 

fi  If  on  each  subregion  It  agrees  with  a  function 

h(i)(t,0)  that  is  on  the  closure  of 

Two  more  assumptions  can  now  be  stated. 

Assumption  2.  The  function  x  (t,©)  maps  Q  in  a 

one-to-one  fashion  onto  a  subregion  of  the  region  in 
(t,x)— space,  and  ^  one-to-one  fashion  onto  an 

n^lmensional  manifold  that  forms  part  of  the  boundary  of 

Assumption  3«  There  exist  functions  t^(©)  that  Induce 

a  regular  decomposition  of  ft  such  that:  (l)  The  function 

u  (t,©)  Is  piecewise  C"  on  ft.  (ll)  If  f,  G,  or  R 

possess  manifolds  of  discontinuity  that  lie  In  '7^  {sls  discussed 

In  Sec.  9),  then  each  of  these  manifolds  Is  coincident  with  the 

Image  of  some  set  1*2,  ... ,  a.  (Hi)  For  each 

component  of  the  constraint  vector  R,  we  have  either 

R'^(t,x*(t,©),u*(t,©) )  »  0  on  ftj^,  or,  with  the  possible 

1  #  # 

exception  of  a  finite  number  of  points,  R*'(t,x  (t,©),u  (t,©))  >  0 
on  ft^ . 
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We  shall  denote  the  image  of  by  1  ■  1,  ,  a, 

and  the  Imeige  of  by  1  ■  2,  ...,  a  +  1.  The 

function  X*  also  maps'^^  onto  whence  we  may  set 

^ '  Note,  however,  that  the  mapping  of  '77^  onto  tX 
in  general  is  not  one— to— one. 

Iff 

lemma  1.  The  function  x  (t,e)  is  continuous  and  is 
piecewise  C"  on  fl.  The  sets  i  ■  1»  •••»  a  +  1,  are 

manifolds  of  class  C" . 

♦  _ 

Let  ^(1)  denote  the  function  that  la  C"  on  and 

* 

that  agrees  with  u  on  Let  denote  the  function 

that  la  C’’  on  ^  x  and  that  agrees  with  G  on  x~6i. 

We  may  extend  the  function  ®  function  that 

has  range  in  ■u  and  that  is  C"  on  a  region  containing 
(and  hence  '~/l^  and  ^2^  interior.  We  may  also 

extend  to  a  function  that  is  C"  on  a  region 

containing  x  in  its  interior.  It  now  follows  from 

(10.5)»  the  properties  of  t^(a)  and  x^(a).  Assumption  5— (l), 
and  standard  theorems  about  the  behavior  of  solutions  of 
differential  equations  with  respect  to  parameters  and  initial 
conditions,  that  x  (t,e)  is  C"  on  Since  ^^2 

given  by  t  -  ^2(6)  and  x  ■  »  x*(t2(0),6),  it  follows 

that  tgC©)  and  x^C©)  are  C".  The  argument  Just  given  can 
be  repeated  with  the  appropriate  modifications  on  and 

with  t  -  ^  “  *2(6)  as  the  boundary  conditions 

for  (10. 5) •  We  then  see  that  x  has  the  desired  properties 
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on  0^  and  Is  continuous  on  v  and  that  is  given 

by  t  =  t^(0),  X  =  x^(0)  a  x*(t^(0),0).  Proceeding  inductively 
in  this  fashion,  we  can  establish  the  desired  properties  for 
X  .  We  note  that  the  sets  i  *  a  +  1,  are  given 

by  functions 

(IQ. 6)  t  =  t^(0),  X  -  Xj,(0)  a  x*(tj^(e),S), 

and  hence  are  manifolds  of  class  C' . 

Assumption  4.  For  every  subregion  0^  at  positive 

distance  from  there  exists  a  positive  constant  d(O^) 

♦  T  1 

such  that  ||xg(t,0)||  ^  d(n  )  on  Q  .  (At  boundary  points 
of  and  at  points  of  ^  1  bounding  away  from 

zero  of  the  determinant  Is  to  be  Interpreted  for  the  various 
limits . ) 

It  C6m  be  shown  that  If  c7"ls  n— dimensional,  then  the 
assumption  that  each  curve  K(6)  Is  not  tangent  to  Implies 
the  existence  of  a  constant  d  >  0  such  that  | |xg(t,0) | |  >  d 
on  all  of 

It  is  an  Immediate  consequence  of  Assumption  4  and  (10.6) 
that  the  manifolds  1  =  2,  ...,  a  +  1,  have  dimension 

n.  It  also  follows  from  Assumption  4  that  the  curves  K(0) 
are  not  tangent  (from  either  side)  to  a  manifold  1^2. 

Prom  Assumption  2,  It  follows  that  on  the  relation 

X  »  X  (t, ®)  can  be  inverted  to  give  a  relation 
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(10.7)  0  -  0{t,x), 


where  d  Is  a  single— valued  function  on  le.  It  further 
follows  from  Lemma  1,  Assumption  4,  and  the  Implicit— function 

theorem  that  9  Is  C”  on  each  7^^,  1  >  2,  and  on  the  set 

^  .  Since  X  (t,9)  I3  one-to-one.  It  follows  that  9 

Is  continuous  on  .  Prom  the  Identity  9  =  9(t,x  (t,9))  It 

follows  that  as  (t,x)  tends  to  c/"  along  K(9),  the  function 
9  will  tend  to  the  value  9.  in  general,  9  will  not  tend 
to  a  unique  limit  at  points  of  t/".  It  can  be  shown,  however, 
that  If  is  n-^lmenslonal  and  the  curves  K(9)  are  not 
tangent  to  (z' ,  then  9  Is  C’’  on  as  well  as  on 

1  >  2. 


Assumption  (l)  For  every  point  (t,x)  ■  (t,x  (t,9)) 

In  ie.  the  control  problem  (2.5)  with  Initial  point  (t,x) 
has  a  unique  solution  In  which  the  optimal  control  Is  u  (t, 9), 
t  <  t  <  t^(9),  and  the  corresponding  curve  is  K(9).  (il) 

There  exists  a  multiplier  vector  ( ^q(  9)  >  ^(  t,  9) ,p.(  t ,  9) )  along 
each  K(9)  such  that  »  1  and  the  functions  ^2^(9)  «  X.(tQ,(9), 9) 
and  M;^(9)  «  M.(tj^(9),9)  are  C  on  -t'. 


The  existence  of  multipliers  along  each  K(9)  follows 
from  Theorem  2;  the  assumption  concerns  the  properties  of 
Xq,Xi,  and 

A  function  u  (t,9)  such  that  Assumptions  1—5  hold  will 
be  called  a  nonmal  parametric  synthesis  of  the  control. 
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Remark.  If  U  la  n— dimensional  and  each  K(6)  Is  not 
tangent  to  %/  ,  then  (11)  follows  from  the  Corollary  of 
Theorem  3  and  the  transversallty  condition  (6.5)* 

Define 

(10i8)  U*(t,x)  =  u*(t, e(t,x) ) . 

*  „ 

It  follows  from  the  preceding  discussion  that  U  is  C  on 
each  for  1  >  2,  and  Is  C"  on  Along  each 

K(0),  however,  U  (t,x)  does  tend  to  a  limit  as  t/  Is  approached. 
If  ^  is  n— dimensional  and  the  curves  K(6)  are  not  tangent 
to  (f,  then  U  Is  C"  on  7^^  as  well.  We  call  U  a 
normal  synthesis  of  the  control. 

11.  THE  FUNCTIONS  X,  |a,  ^  L 

Lemma  2.  The  functions  X(t,e)  and  M-(t,e)  are  piece- 
wise  C  on  fi.  Across  every  manifold  1-2,  ...,  a, 

equation  (9.1)  holds.  If  f,  G,  and  R  are  continuous  across 
then  so  Is  X(t,6). 

Let  6  denote  the  vector  formed  by  taking  those  components 

R*^  of  R  such  that  R'^(t,x  (t,G),u  (t,©))  *0  on  Let 

It  be  the  vector  obtained  from  u  by  taking  the  corresponding 

components.  Prom  (2.2)— (ii).  It  follows  that  R^  has  maximum 

rank,  say  on  Let  be  an  r^^  by  r^^  nonsingular 

submatrix  of  R  .  Let  denote  the  vector  obtained  from 

u  u 

by  selecting  the  components  corresponding  to  the  columns  of 
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used  to  obtain  R^.  In  order  to  simplify  the  exposition 
we  shall  assume  that  the  same  submatrix  Is  nonsingular  at  all 
points  of  It  will  be  seen  from  the  ensuing;  discussion 

that  this  restriction  can  be  easily  overcome. 

Prom  (6.3 )»  we  get 

(11.1)  -  -  (Hq)(R^)“^. 

Since  by  (6.4)  those  components  of  p.  that  are  not  included 
In  |1  vanish  on  we  may  write  (6.2)  along  each  K(6)  as 

follows ; 

(11.2)  x^(t,e)  -  -  +  (H-)(R^)“^Rx^  Mti,e)  -  x^(e), 

where  the  arguments  of  the  functions  on  the  right-hand  side  are 

*  # 

(t,x  (t,0),u  (t,0)).  A  proof  similar  to  that  used  In  Lemma  1 
can  now  be  used  to  show  that  X(t,e)  Is  of  class  C'  on 
It  then  follows  from  (11.1)  that  p  Is  also  C'  on  Since 

the  other  components  of  p  vanish  on  the  vector  p  is 

C  on 

A  A 

The  same  arguments  applied  to  with  R-,  and 

p  appropriately  redefined  and  with  the  proper  Initial  data 
X(t2(0)i®)i  show  that  X(t,e)  and  p(t,6)  are  C'  on 
The  initial  data  X(t2(6)>6)  are  defined  by  continuity  or  by 
(9.1)  If  ^2  corresponds  to  a  manifold  of  discontinuity  of 
f,  G,  or  R.  Proceeding  backwards  in  this  fashion,  we  see 
that  X  and  p  are  piecewise  C  on  n  and  have  the  requisite 
continuity  properties. 
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Define 

(11.5)  L(t,x)  =  X(t,0(t,x)),  (t,x)  eT^,  1  =  1,  a. 

we  list  the  properties  of  L(t,x)  In  the  following  Lemma: 

Lemma  3 »  The  function  L  ^  C  on  each 
1  :»  2,  a,  and  on  the  set  .  Moreover,  If  f,  G, 

and  R  are  continuous  across  a  manifold  ,  1  »  2 ,  . . . ,  a , 
then  so  Is  L.  Across  a  manifold  (9*1)  holds  with  X 

replaced  by  L,  where  +  now  Indicates  a  limit  from  the 
Interior  of  3-nd  —  Indicates  a  limit  from  the  Interior 

of  .  IX  I3  n— dimensional  and  the  curves  K( 0 )  are  not 
tangent  to  then  L  1^  C  on  also. 

The  proof  of  this  lemma,  except  for  the  next  to  the  last 

sentence.  Is  an  Immediate  consequence  of  Lemma  2  and  the 

properties  of  9(t,x).  It  Is  clear  from  the  properties  of  f, 

G,  X(t,0),  X  (t,0),  and  u  (t,©)  that  H(t,x  (t, 0), X(t, 6),u  (t,0 

Is  continuous  on  each  of  the  sets  U  and  ^ 

* 

1=2,  a.  Hence,  H{t,x,L, U  )  Is  continuous  on  each 

U  and  Rj__^  U  1  -  2,  ...,  a.  If  y7[^  Is  not 

a  manifold  of  discontinuity  of  f,  G,  or  R,  then  by  Theorem  2, 
H  Is  continuous  across  along  each  K(0).  Hence  from  the 

continuity  of  H  on  R^  U  and  U  75^^  It  follows 

that  H  Is  continuous  across  unrestrictedly  In  this 

case.  Since  L  Is  continuous  across  7^,  (9.1)  holds  across 

unrestrictedly.  A  similar  argument  shows  that  If 
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Is  a  manifold  of  discontinuity  of  f,  0  or  R,  then  (9«1) 
holds  across  also  without  the  restriction  that  the  limits 

be  taken  along  K(0). 

12.  THE  VALUE  AND  THE  HAMILTON-JACOBI  EQUATION 

Let  Assumptions  1  to  5  of  Sec.  10  hold.  Then  we  can  define 
a  function  W(t,x)  on  by  assigning  to  each  point  (t,x) 

In  7^ the  value  that  the  functional  (2.>)  with  (tQ^x^)  »  (t,x) 
takes  along  the  optimal  curve  K(S)  through  (t,x).  Thus  we 
have 

(12.1)  W(t,x)  -  W(t,x*(t,©)) 

-  g(<j)  +  J  f(t,x*(t,d),u  (t,0))dt, 

t 

where  9  and  o  are  related  by  (10.2).  We  shall  call  W 
the  value  function,  or  simply  the  value  of  the  control  problem. 
We  summarize  the  properties  of  W  In  the  following  theorem: 

Theorem  4 .  The  value  VJ  Is  continuous  on  ■7^  ^  C  " 
on  each  1  >  2,  and  Is  C''  on  ^  •  On  each 

1  a  1,  ...,  a,  we  have 

W^(t,x)  -  -  f(t,x,U*(t,x))  -  L(t,x)0(t,x,y*{t,x)), 

(12.2) 

W^(t,x)  -  L(t,x). 

At  points, of  a  manifold  ,  1  >  2 ,  ( 12 . 2 )  holds  for  the 

one-^slded:  limits.  If  Is  not  a  manifold  of  discontinuity 
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of  f. 


G,  or  R,  then-  and  are  continuous  across 

Across  every  manifold  J  =•  2,  a,  the  relation 


'^t  '^x  \ 


holds  for  all  differentials 


along  • 


Remark  1.  If  we  substitute  the  second  equation  of  (12.2) 


into  the  first,  we  see  that  the  value  satisfies  the  Hamllton- 


Jacobi  equation  on  each 


Remark  2 .  It  follows  from  the  properties  of  L,  U  ,  and 
0  that  both  W.  and  W  possess  limits  as  (t,x)  tends  to 
^  along  a  curve  K(0),  even  though  in  general  W+.  and  ^ 

w  X 

do  not  possess  limits  as  (t,x)  tends  to  «/.  If,  however, 
iT  is  n-dlmenslonal  and  the  curves  K(6)  are  hot  tangent  to 
e/  ,  then  W  is  C"  on 


Remark  3 •  In  Assumption  5— (li)  we  supposed  that  along 
each  K(6)  there  was  one  set  of  multipliers  with  »  1. 

The  second  equation  in  (12.2)  now  shows  that  if  there  is  one 
such  set  satisfying  the  other  requirements  of  Assumption  5, 
then  it  must  be  unique. 

The  proof  that  we  now  give  for  Theorem  4  is  an  extension 
of  an  argument  used  in  the  calculus  of  variations  to  prove 
the  Invariance  of  Hilbert's  Integral  in  certain  fields. 

It  is  clear  from  (12.1)  that  W  is  continuous.  Let 


(12.3) 


t  -  Tq(s),  X  -  Xq(s):, 


Q  <  s  <  1, 
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define  a  curve  F  that  does  not  Intersect  Itself  and  that> 
with  the  possible  exception  of  end  points,  lies  entirely  within 
some  definiteness  we  take  1  -  a.  It  follows  from 

Assumptions  2  and  4  that  the  system  of  equations 


(12.4)  Tq(s)  -  t,  Xq{s)  -  x*(t,e),  0  <  s  <  1, 


defines  a  function  B  »  0(s)  that  Is  C'^  on  [0,1]  •  Hence 
If  we  traverse  P  as  s  goes  from  0  to  1,  we  obtain  a 
family  of  curves  K(s)  -  K(©(3))  by  means  of  the  function 
x*(t,©(3)),  where  Tq(s)  <  t  <  t^(s).  Since  the  manifolds 

J  «  1,  a,  are  given  by  (10.6),  It  follows  that  the 


3 


Intersections  of  the  curves  K(8)  with  th®  manifolds 

are  given  by 

X  »  Tj(s)  a  tj(e(s)). 


(12.5) 


X  =  Xj(s)  s  x*(tj(e(s)),e(s)). 


J  ■  1  ^  •  •  •  >  Gt  • 


The  functions  Tj 
C"  on  [0,1].  For  J  = 
from  (12,5)  In  two  ways. 


(12.6) 


ds  ■ t  ds 


and  Xj ,  J 
2  ,,  .  » • ,  Ot  > 


+  X 


*+  ^ 
6  ds 


»  2,  ...,  a,  are  clearly 
we  can  compute  dXj/ds 


^t  ds 


♦ _ 

+ 


ds 


S 


where  the  superscript  +  Indicates  that  we  are  taking  limits 
from  the  Interior  of  and  the  superscript  -  Indicates 

limits  from  the  Interior  of  7^.  Equation  (12.6)  also  holds 
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for  J  =»  0,  without  the  superscripts  +  and  — .  Prom  (12.5)# 
(10.2)  —  (10.5)#  and  standard  theorems  on  the  differentiation 
of  solutions  of  differential  equations  with  respect  to  initial 
data#  we  get 

dT 

ds”  “  ^la  df^  ) )  »  (-  M,  O), 

where  M  is  the  matrix  (8.1)  and  0  is  the  n  by  (n  —  p) 
zero  matrix. 

We  now  consider  W  along  T.  Prom  (12.1)  we  obtain 

W(TQ(a),XQ(3))  = 

.Ti(3) 

g(a(3))  f(t,x*(t,e(s))#u*(t,e(s)))dt. 

To(s) 


Hence  dw/ds  exiata  and  la  given  by 


#  # 

where  df/3a  ■  (f^  x^  +  f^  Ug)(dS/ds),  the  superacrlpts  + 
and  —  have  the  aame  meaning  as  In  ( 12 . 6 )  #  and  the  ax^uments 
of  the  functions  are  (t,0(s)).  Prom  (6.2)  we  get 
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f  »  -  (X.  +  XG^  +  liR  ),  and  from  (6.3)  we  have 

X  U  X  X 

a  —  (XQ^  +  *  Hence  we  obtain 

M  “  [^t  ^0 


(12.9) 


+ 


* 


+  R, 


u 


ds 


The  components  of  the  vector 
written  as  follows: 


|i(R^  Xq  + 


R. 


u«) 


u  “0 


can  be 


r 

(12.10)  2 

k-1 


X* 


0-' 


If  *  V 

R.  u  ,  )  s 


u 


e-* 


r 

2 

k»l 


k  ^ 
d0^ 


li 


n. 


If  at  a  point  (t,0)  In  Q,  we  have  R^(t,x* ( t, 0) ,u* (t, 0) )  >  0 
then  from  (6.4)  we  obtain  |J.^(t,0)  -  0.  On  the  other  hand, 
since  R‘"(t,x*(t,0),u*(t,0))  >  0  on  If  R*^  -  0  at  (t,0) 

then  R^,  as  a  function  of  {t,0),  has  a  minimum  at  this  point 
Since  (t,0)  is  interior  to  fl,  SrV^®^  vanishes  at  this 
point  for  all  1-1,  ...,  n.  Hence  (12.10)  Is  zero  for  all 
(t,0). 

If  we  set  aG/50  =  (G^^  +  G^  u*),  then  from  (10.5)  we 

■»  *  *  . 
have  x^g  =  ^G/90.  Furthermore,  we  have  x^g  ■  ^0t’  Hence  we 

may  write  (12.9)  as 


3  f  -  _  (  X  x  "'!  ^ 

^0^^  ds  * 


Substituting  this  expression  into  the  Integral  In  (12.8)  and 
performing  the  integration  gives 
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If  we  now  use  (12.6),  (12.7),  and  the  relation  «  G,  and 
substitute  the  resulting  expression  Into  (12.8),  using  the 
definition  of  H,  we  get 


ds 


-I  to(s  ) 


+  2  (H"  -  H"^)  -  (X~  -  X+) 

>2  L'  '  d3 


Prom  (6.5)  we  see  that  the  first  square  bracket  vanishes.  Prom 

Theorem  2  and  (9«1)  It  follows  that  every  square  bracket  In 
a 

Z  vanishes.  Hence,  since  F  Is  arbitrary,  we  have 
J-2 


(12.11)  dW  -  -  H  dT  +  L  dX 


for  arbitrary  differentials  (dT,  dX) .  The  theorem  Is  an 
Immediate  consequence  of  (12.11),  the  properties  of  f,  G,  L, 
and  U  ,  Theorem  2,  and  (9*1) • 

13.  AN  EQUATION  OP  DYNAMIC  PROGRAMMING 

Por  each  (t,x)  in  let  denote  the  set 

of  admissible  controls  u  at  (t,x}*  Since  U  is  a  normal 
synthesis,  it  follows  from  (6.6)  that  for  any  (t,x)  in  ^ 

we  have 


I  Q , 
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(15.1)  H(t,x,L(t,x)U*(t,x))  -  min  H(t,x,L(t,x)u) . 

u  €  a(t,x) 

If  we  apply  (12.2)  to  (I3. 1),  we  see  that  on 


(13.2)  -  - 


min 


(f(t,x,u)  +  W^O(t,x,u)]. 


u  €(3.(t,x) 

If  (tjx)  lies  on  a  manifold  1-1,  a,  then  the 

relations  (I3.I)  and  (I3.2)  hold  for  the  one— sided  limits. 

Equation  (I3.2)  Is  the  functional  equation  obtained 
formally  by  Bellman  [l]  for  control  problems  In  which  t/”  is 
the  n-dlmenslonal  manifold  tj^  «  constant  and  f,  Q,  and  R 
are  C".  We  note  that  (13*2)  holds  for  more  general  problems 
than  these.  Since  (13*1)  Is  a  restatement  of  the  Welerstrass 
condition,  since  (12.2)  says  that  on  each  W  satisfies 

the  Hamllton-Jacobl  equation,  and  since  Pontryagln's  principle 
derives  from  the  Welerstrass  condition,  the  relationship  between 
these  Items  and  (I3.2)  Is  clear. 

We  remark  that  computational  schemes  based  directly  on 
(13.1)  In  the  case  that  t/  Is  of  dimension  p,  with  p  <  n, 
will  encounter  difficulties  because.  In  general,  W^.  and  W„ 


do  not  exist  at  v  .  (See  Remark  2,  Theorem  4.) 


14.  THE  PROBLEM  OP  SYNTHESIS 

«  # 

I^t  u  (t,6)  and  x  (t,R)  be  as  In  Assumptions  1  to  4, 
and  let  us  replace  Assumption  5  by  the  following: 

Assumption  6.  Along  each  K(R),  let  equations  (6.1)  — 
(6.5)  hold  with  ^0  “  ^(b,6)  and  Mr(t>6)  have 
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the  properties  described  In  Theorem  2.  Let  the  functions 

and  be  as  In  Assumption  3«  Let  the  function  H  be  such 

that  (9*1)  holds  for  all  manifolds  J  ■  2,  a. 

Assumption  6  consists  of  those  consequences  of  Assumption 
5  that  enter  Into  the  discussion  of  Sec.  10.  Hence,  If  we  now 
look  upon  W  as  being  defined  by  (12.1),  then  Theorem  4  still 
holds.  In  particular,  (12.11)  holds.  Moreover,  If  we  take  P 
to  lie  entirely  on  a  manifold  1-2,  ...,  a  +  1,  then 

the  arguments  used  to  establish  (12.11)  for  T  In  some 
show  that  (12.11)  holds  for  T  on  a  manifold  1  2. 

For  curves  F  on  a  validity  of  (12.11)  follows 

from  (6.5) •  Hence  the  Integral 


(14.1)  ^  H(t,x,L  U*)dT-  L(t,x)dX 

r 

Is  Independent  of  path  In  i^ 

a  finite  number  of  arcs,  each  arc  lying  entirely  In  some 
or  on  a  manifold  1-1*  ...,  a. 

From  the  preceding  discussion  we  see  that  Assumptions  1 

to  4  and  Assumption  6  determine  for  the  control  problem  the 

analogue  of  a  field  In  the  calculus  of  variations,  with  (14.1) 

as  the  Hilbert  Invariant  Integral.  The  following  theorem  can 

now  be  established  by  using  the  same  argument  as  Is  used  for 

the  analogous  fundamental  sufficiency  theorem  In  the  calculus  of 

variations : 
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Theorem  5.  Let  Assumptions  1  to  4  and  Assumption  6  hold. 

Furthermore j  let  (6.6)  hold  on  '76'  for  u*  =  U(t,x).  Then 
* 

u  (t,0)  I3  a  normal  parametric  synthesis  of  the  control  and 
U  (t,x)  Is  a  normal  synthesis  of  the  control. 
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